The U.S. were trailing Britain after three legs when Alexis Holmes initially failed to collect the baton from Quanera Hayes and by the time she got hold of it she appeared to have gone beyond the "blue box".
The U.S. recovered to finish second behind the British in the heat but were marked as disqualified.
Jamaica, perennial global silver medallists in recent years, won the second heat in 2:59.82, with fast-finishing France and Italy following them through.
Fifth-placed Belgium, bronze medallists in the last two worlds, did not make it.
Reporting by Mitch Phillips, editing by Ken FerrisOur Standards: The Thomson Reuters Trust Principles.
Persons:
Alexis Holmes, Quanera Hayes, Sarah Meyssonnier, Justin Robinson, Rajesh Ramesh, Mitch Phillips, Ken Ferris
Organizations:
Athletics Centre, Rights, United, Saturday's, Fifth, Thomson
Locations:
Budapest, Hungary, U.S, United States, Jamaica, Canada, India, Botswana, France, Italy, Belgium